Techniques for data retention

ABSTRACT

Systems and techniques for managing data in a relational database environment and a non-relational database environment. Data in the relational database environment that is static and to be maintained beyond a preselected threshold length of time is identified. The data is copied from the relational database and stored in the data the non-relational database. Access to the data is provided from the non-relational database via a user interface that accesses both the relational database and the non-relational database.

CLAIM OF PRIORITY

This application is related to, and claims priority to, provisionalutility application No. 61/905,460 entitled “FIELD HISTORY RETENTION,”filed Nov. 18, 2013, and having attorney docket No. 1366PROV;provisional utility application No. 61/904,822 entitled “SCALABLEOBJECTS,” filed on Nov. 15, 2013, and having attorney docket No.1362PROV; provisional utility application No. 61/904,826 entitled“MULTI-TENANCY FOR A NOSQL DATABASE,” filed Nov. 15, 2013, and havingattorney docket No. 1363PROV; provisional utility application No.61/905,439 entitled “BIG OBJECTS,” filed Nov. 18, 2013, and havingattorney docket No. 1364PROV; and provisional utility application No.61/905,457 entitled “ORCHESTRATION BETWEEN TWO MULTI-TENANT DATABASES,”filed Nov. 18, 2013, and having attorney docket No. 1365PROV, the entirecontents of which are all incorporated herein by reference.

TECHNICAL FIELD

Embodiments relate to techniques for storage and management of data.More particularly, embodiments relate to selectively retaining databased on preselected characteristics.

BACKGROUND

Relational databases are commonly used to store large amounts of data.Current relational database systems have limits beyond which the systemdoes not scale well. Thus, environments in which large amount of datamust be managed (e.g., on-demand services environments, multitenantdatabase environments, electronic commerce, logistics) may near or reachconditions in which the relational database becomes less effective.Accordingly, current relational database management systems are notideal in some situations.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are illustrated by way of example, and notby way of limitation, in the figures of the accompanying drawings inwhich like reference numerals refer to similar elements.

FIG. 1 is a block diagram of one embodiment of an architecture that mayprovide data retention as described herein.

FIG. 2 is a flow diagram of one embodiment of a technique fortransferring data from a relational database environment to anon-relational database environment.

FIG. 3 is an interaction diagram of one embodiment of a technique forquerying a non-relational (NoSQL) database using relational database(SQL) commands.

FIG. 4 is a block diagram of one environment wherein an on-demanddatabase service might be used.

FIG. 5 is a block diagram of one embodiment of elements of environmentof FIG. 4 and various possible interconnections between these elements.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth.However, embodiments of the invention may be practiced without thesespecific details. In other instances, well-known circuits, structuresand techniques have not been shown in detail in order not to obscure theunderstanding of this description.

As discussed above, there are conditions and/or situations in which arelational database system is being used to manage large amounts of datathat does not require the functionality of a relational database system,but is related to data that does. The examples that follow are presentedin terms of field history and field history archives; however, thetechniques described herein are not limited to these examples.

When records are edited or modified in a relational databaseenvironment, entries corresponding to the change can be made. Forexample, an initial account record may be:

Account ID Account Name Expected Value 123456789 Acme, Inc. $1,000,000The account record can then me modified, for example, to:

Account ID Account Name Expected Value 123456789 Acme Systems, Inc.$2,000,000The corresponding field history information can be maintained in anothertable:

Field History ID Parent ID Old Value New Value 0000000001 123456789Acme, Inc. Acme Systems, Inc. 000000002 123456789 $1,000,000 $2,000,000In this simplified example, for each column in the original object thatis changed a row is generated in the history table. The history tablecan be useful for keeping an audit trail, for example

With an active environment, the history can grow to a very large numberof rows quickly. However, because the data is static after creation andthe data is not frequently accessed, the features and flexibilityprovided by a relational database system may be unused. Thus, adifferent, more scalable database structure can be utilized for thesetypes of data.

In one embodiment, this type of information can be stored on anon-relational database, for example, Apache HBase, which is an opensource non-relational distributed database. Other databases can also besupported. In one embodiment, a JAVA® Database Connectivity (JDBC)driver can be utilized to support low-latency SQL queries to run overthe data stored in the non-relational database (e.g., HBase).

A non-relational database can provide better horizontal scalability thana relational database model and provide linear access characteristics,and simpler read and write semantics. In one embodiment, one or moreHBase databases can be integrated into a platform (e.g., salesforce)using a framework or interface (e.g., External Objects in salesforce)that allows for data from external sources to be utilized in theplatform. In one embodiment, the framework/interface allows for theexternal database/resource to appear to a user as if it were a part ofthe platform.

In one embodiment, transfer of data from the relational databaseenvironment to the non-relational database environment is notcontinuous. Transfer of data can occur in response to a request and/orin response to a set of one or more preselected conditions, which caninclude, for example, exceeding a threshold number of rows in a table,exceeding a threshold database size, and/or available bandwidth. Otherconditions can also be used.

In one embodiment, the techniques described herein are provided within amultitenant database environment. Within a multitenant databaseenvironment, the conditions that trigger transfer or data and/or theformat of the data may vary from tenant to tenant. In one embodiment,each tenant of the multitenant database can have a custom interface thatcan be utilized to access information in the relational databaseenvironment as well as the non-relational database environment.

In one embodiment, the functionality described herein operates toprovide a query agent with a JDBC application programming interface(API) from the perspective of a client device. The query agent operatesto translate a SQL query (e.g., passed through as a string in the JDBCAPI) into a series of “native” NoSQL store APIs. In one embodiment, theAPI to the NoSQL store is at a lower level, so the techniques describedherein allow a higher-level query language (e.g., SQL) to be used toread/write data and manage schemas. Various architectures are providedin the description that follows.

FIG. 1 is a block diagram of one embodiment of an architecture that mayprovide data retention as described herein. In one embodiment, clientdevices are used by one or more users to access services from a serviceprovider. The service provided can be, for example, an on-demandservices environment, a multitenant database environment, or any othertype of service provider.

Client devices 110 and 115 operate to allow a user to access remoteservices provided by service provider 140 via network 130. Clientdevices 110 can be, for example, desktop computers, laptop computers,tablets, smart phones, thin clients, etc. Network 130 can be anynetwork, for example, the Internet, a corporate local area network orwide area network, a cellular network, and/or any combination thereof.

Service provider 140 can be any number of servers and/or other devicesthat operate to provide services to one or more client devices. In oneembodiment, service provider 140 operates with one or more relationaldatabases (e.g., 150) and one or more non-relational databases (e.g.,160). Service provider 140 operates using relational database 150 andnon-relational database 160 as described above.

In one embodiment, service provider 140 is an on-demand servicesenvironment with multiple client organizations that provides differentand/or different levels of services to the client organizations. Forexample, service provider 140 can be a multitenant database environmentthat provides custom interfaces and data isolation to the differentclient organizations. In the example, multitenant database environment,the transfer of data from relational database 150 and non-relationaldatabase 160 can be on an organization-by-organization basis withdifferent parameters and/or conditions for different organizations.

FIG. 2 is a flow diagram of one embodiment of a technique fortransferring data from a relational database environment to anon-relational database environment. Data to be moved from therelational database environment to the non-relational databaseenvironment is identified, 210. Various parameters and conditions areused to determine what data is to be moved/copied/transferred.

In a multitenant database example, not all organizations/tenants mayhave the functionality to copy data from the relational database to thenon-relational database as described. That is, the functionality may beprovided on a tenant-by-tenant basis. Further, the fields and/or datathat can be copied can be limited/determined/etc. on a tenant-by-tenantbasis. In one embodiment, the data to be copied for a particular tenantis based on a key prefix and/or date (e.g., field history, older than aspecified date).

The selected data is copied form the relational database to thenon-relational database, 220. In one embodiment, a message is enqueuedwith the parameters (e.g., field history, older than a specified date)for the data to be copied. In one embodiment, the message is used tohandle copying of the data in batches. For example, when a chunk of rowshave been processed, the message (or a variation) is enqueued again tohandled the next chunk of rows until all of the specified data has beencopied.

In one embodiment, a chunk of data to be copied is delineated by a datarange and an offset. The offset is used to prevent re-loading of rowsthat have already been copied. In one embodiment, when the selected datahas been copied, a message handler marks the current job as successfuland can insert a row in the source table to help track what data hasbeen copied. Data can then be deleted from the relational database.

Access to the copied data is then provided with a common user interface,230, so that the user has access to the copied data that is stored inthe non-relational database environment. In one embodiment, theinterface providing access to the relational database environment alsoincludes virtual entity or other interface to the non-relationaldatabase to allow the user seamless access to data copied from therelational database environment to the non-relational databaseenvironment.

In contrast to turning SQL queries into batch-oriented map/reduce jobs,the techniques described herein can be utilized to transform the SQLqueries into a set of HBase (or other non-relational database) scansthat can be executed in parallel for each row key range. In oneembodiment, these scans are executed in parallel for each row key rangeand can be combined to provide results of the query. As a result, thelatency of the queries is low enough to allow data to driveanalytic-type queries over large amounts of data. In one embodiment, allthis is hidden behind a JDBC driver. In one embodiment, the userprovides a schema for their database table data and a SQL query. In oneembodiment, column values can be be mapped to individual KeyValuesand/or combined together in a single KeyValue to reduce the size ofdata, which can improve read speed.

FIG. 3 is an interaction diagram of one embodiment of a technique forquerying a non-relational (NoSQL) database using relational database(SQL) commands. In one embodiment, the technique of FIG. 3 is performedwithin a multitenant database environment.

SQL interface 310 is any type of interface/client device that can beused to receive SQL commands and provide results form the SQL commands.For example, SQL interface 310 can be a SQL application running on aclient computing device. SQL-to-NoSQL agent 320 provides thefunctionality described herein. SQL-to-NoSQL agent 320 may be acentralized single agent or can be distributed over multiple entities.Non-relational database 330 can be any type of non-relational database,for example, HBase.

In response to receiving at least one SQL command representing a query,SQL interface 310 sends the query, 350, to SQL-to-NoSQL agent 320. Inresponse to receiving the SQL command, SQL-to-NoSQL agent 320 parses thequery, 352. SQL-to-NoSQL agent 320 then compiles a query, which caninclude retrieving metadata, 354, from non-relational database 330. Thequery plan can be optimized, 356. In one embodiment the SQL query istransformed into one or more scans that are relatively simple, forexample, with no joins, basic filtering and/or simple aggregation.

In one embodiment, the scans can be run on a sub-section of tables sothat not all tables need to be replicated in the non-relationaldatabase. In some embodiments, the results need only be approximatelycorrect. Other optimizations can be utilized to provide the desiredlevel of performance.

The query plan can be executed as multiple parallel scans, 360, ofnon-relational database 330. In one embodiment, a set of HBase (or othernon-relational database) scans that can be executed in parallel for eachrow key range. In one embodiment, these scans are executed in parallelfor each row key range and can be combined to provide results of thequery.

In one embodiment, non-relational database 330 can perform filteringand/or aggregation. Results of the multiple parallel scans are returned,365, to SQL-to-NoSQL agent 320. In one embodiment, SQL-to-NoSQL agent320 can perform merge sorting on the results. By combining the resultsof the one or more scans, the system can provide an aggregated/unifiedresult to the original SQL query. The results are provided, 370, to SQLinterface 310.

In one embodiment, deletion from the relational database environment isdecoupled from the copy process. In embodiment, a system job in therelational database environment periodically (e.g., daily, weekly, 12hours) runs to query tenants/organizations that have the functionalitydescribed herein enabled to determine whether any data copy jobs havebeen completed. If so, the data that has been copied to thenon-relational database environment may be deleted from the relationaldatabase environment.

In one embodiment, when a deletion message/job is processed, the handlerdetermines the parameters (e.g., field history, older than a specifieddate) for the deletion request. In one embodiment, this is accomplishedwith a system-level job. It can, for example, fun a query to find allorganizations that have post-archival deletion enabled and havepost-archival deletion requests that have not been processed. In oneembodiment, the system-level job can enqueue a message for eachcombination. In one embodiment, other job types (e.g., anorganization-level job) can be utilized to process deletions.

The non-relational database can be queried to determine the data withinthe specified range. For each chunk, the handler passes identifiersloaded from the non-relational database environment to the relationaldatabase environment to cause a hard delete of the corresponding rowsfrom the relational database environment. Loading the identifiers fromthe non-relational database environment to the relational databaseenvironment ensures that data will not be deleted before beingsuccessfully copied from the relational database environment to thenon-relational database environment.

FIG. 4 illustrates a block diagram of an environment 410 wherein anon-demand database service might be used. Environment 410 may includeuser systems 412, network 414, system 416, processor system 417,application platform 418, network interface 420, tenant data storage422, system data storage 424, program code 426, and process space 428.In other embodiments, environment 410 may not have all of the componentslisted and/or may have other elements instead of, or in addition to,those listed above.

Environment 410 is an environment in which an on-demand database serviceexists. User system 412 may be any machine or system that is used by auser to access a database user system. For example, any of user systems412 can be a handheld computing device, a mobile phone, a laptopcomputer, a work station, and/or a network of computing devices. Asillustrated in herein FIG. 4 (and in more detail in FIG. 5) user systems412 might interact via a network 414 with an on-demand database service,which is system 416.

An on-demand database service, such as system 416, is a database systemthat is made available to outside users that do not need to necessarilybe concerned with building and/or maintaining the database system, butinstead may be available for their use when the users need the databasesystem (e.g., on the demand of the users). Some on-demand databaseservices may store information from one or more tenants stored intotables of a common database image to form a multi-tenant database system(MTS). Accordingly, “on-demand database service 416” and “system 416”will be used interchangeably herein. A database image may include one ormore database objects. A relational database management system (RDMS) orthe equivalent may execute storage and retrieval of information againstthe database object(s). Application platform 418 may be a framework thatallows the applications of system 416 to run, such as the hardwareand/or software, e.g., the operating system. In an embodiment, on-demanddatabase service 416 may include an application platform 418 thatenables creation, managing and executing one or more applicationsdeveloped by the provider of the on-demand database service, usersaccessing the on-demand database service via user systems 412, or thirdparty application developers accessing the on-demand database servicevia user systems 412.

The users of user systems 412 may differ in their respective capacities,and the capacity of a particular user system 412 might be entirelydetermined by permissions (permission levels) for the current user. Forexample, where a salesperson is using a particular user system 412 tointeract with system 416, that user system has the capacities allottedto that salesperson. However, while an administrator is using that usersystem to interact with system 416, that user system has the capacitiesallotted to that administrator. In systems with a hierarchical rolemodel, users at one permission level may have access to applications,data, and database information accessible by a lower permission leveluser, but may not have access to certain applications, databaseinformation, and data accessible by a user at a higher permission level.Thus, different users will have different capabilities with regard toaccessing and modifying application and database information, dependingon a user's security or permission level.

Network 414 is any network or combination of networks of devices thatcommunicate with one another. For example, network 414 can be any one orany combination of a LAN (local area network), WAN (wide area network),telephone network, wireless network, point-to-point network, starnetwork, token ring network, hub network, or other appropriateconfiguration. As the most common type of computer network in currentuse is a TCP/IP (Transfer Control Protocol and Internet Protocol)network, such as the global internetwork of networks often referred toas the “Internet” with a capital “I,” that network will be used in manyof the examples herein. However, it should be understood that thenetworks that one or more implementations might use are not so limited,although TCP/IP is a frequently implemented protocol.

User systems 412 might communicate with system 416 using TCP/IP and, ata higher network level, use other common Internet protocols tocommunicate, such as HTTP, FTP, AFS, WAP, etc. In an example where HTTPis used, user system 412 might include an HTTP client commonly referredto as a “browser” for sending and receiving HTTP messages to and from anHTTP server at system 416. Such an HTTP server might be implemented asthe sole network interface between system 416 and network 414, but othertechniques might be used as well or instead. In some implementations,the interface between system 416 and network 414 includes load sharingfunctionality, such as round-robin HTTP request distributors to balanceloads and distribute incoming HTTP requests evenly over a plurality ofservers. At least as for the users that are accessing that server, eachof the plurality of servers has access to the MTS' data; however, otheralternative configurations may be used instead.

In one embodiment, system 416, shown in FIG. 4, implements a web-basedcustomer relationship management (CRM) system. For example, in oneembodiment, system 416 includes application servers configured toimplement and execute CRM software applications as well as providerelated data, code, forms, webpages and other information to and fromuser systems 412 and to store to, and retrieve from, a database systemrelated data, objects, and Webpage content. With a multi-tenant system,data for multiple tenants may be stored in the same physical databaseobject, however, tenant data typically is arranged so that data of onetenant is kept logically separate from that of other tenants so that onetenant does not have access to another tenant's data, unless such datais expressly shared. In certain embodiments, system 416 implementsapplications other than, or in addition to, a CRM application. Forexample, system 416 may provide tenant access to multiple hosted(standard and custom) applications, including a CRM application. User(or third party developer) applications, which may or may not includeCRM, may be supported by the application platform 418, which managescreation, storage of the applications into one or more database objectsand executing of the applications in a virtual machine in the processspace of the system 416.

One arrangement for elements of system 416 is shown in FIG. 4, includinga network interface 420, application platform 418, tenant data storage422 for tenant data 423, system data storage 424 for system data 425accessible to system 416 and possibly multiple tenants, program code 426for implementing various functions of system 416, and a process space428 for executing MTS system processes and tenant-specific processes,such as running applications as part of an application hosting service.Additional processes that may execute on system 416 include databaseindexing processes.

Several elements in the system shown in FIG. 4 include conventional,well-known elements that are explained only briefly here. For example,each user system 412 could include a desktop personal computer,workstation, laptop, PDA, cell phone, or any wireless access protocol(WAP) enabled device or any other computing device capable ofinterfacing directly or indirectly to the Internet or other networkconnection. User system 412 typically runs an HTTP client, e.g., abrowsing program, such as Microsoft's Internet Explorer browser,Netscape's Navigator browser, Opera's browser, or a WAP-enabled browserin the case of a cell phone, PDA or other wireless device, or the like,allowing a user (e.g., subscriber of the multi-tenant database system)of user system 412 to access, process and view information, pages andapplications available to it from system 416 over network 414. Each usersystem 412 also typically includes one or more user interface devices,such as a keyboard, a mouse, trackball, touch pad, touch screen, pen orthe like, for interacting with a graphical user interface (GUI) providedby the browser on a display (e.g., a monitor screen, LCD display, etc.)in conjunction with pages, forms, applications and other informationprovided by system 416 or other systems or servers. For example, theuser interface device can be used to access data and applications hostedby system 416, and to perform searches on stored data, and otherwiseallow a user to interact with various GUI pages that may be presented toa user. As discussed above, embodiments are suitable for use with theInternet, which refers to a specific global internetwork of networks.However, it should be understood that other networks can be used insteadof the Internet, such as an intranet, an extranet, a virtual privatenetwork (VPN), a non-TCP/IP based network, any LAN or WAN or the like.

According to one embodiment, each user system 412 and all of itscomponents are operator configurable using applications, such as abrowser, including computer code run using a central processing unitsuch as an Intel Pentium® processor or the like. Similarly, system 416(and additional instances of an MTS, where more than one is present) andall of their components might be operator configurable usingapplication(s) including computer code to run using a central processingunit such as processor system 417, which may include an Intel Pentium®processor or the like, and/or multiple processor units. A computerprogram product embodiment includes a machine-readable storage medium(media) having instructions stored thereon/in which can be used toprogram a computer to perform any of the processes of the embodimentsdescribed herein. Computer code for operating and configuring system 416to intercommunicate and to process webpages, applications and other dataand media content as described herein are preferably downloaded andstored on a hard disk, but the entire program code, or portions thereof,may also be stored in any other volatile or non-volatile memory mediumor device as is well known, such as a ROM or RAM, or provided on anymedia capable of storing program code, such as any type of rotatingmedia including floppy disks, optical discs, digital versatile disk(DVD), compact disk (CD), microdrive, and magneto-optical disks, andmagnetic or optical cards, nanosystems (including molecular memory ICs),or any type of media or device suitable for storing instructions and/ordata. Additionally, the entire program code, or portions thereof, may betransmitted and downloaded from a software source over a transmissionmedium, e.g., over the Internet, or from another server, as is wellknown, or transmitted over any other conventional network connection asis well known (e.g., extranet, VPN, LAN, etc.) using any communicationmedium and protocols (e.g., TCP/IP, HTTP, HTTPS, Ethernet, etc.) as arewell known. It will also be appreciated that computer code forimplementing embodiments can be implemented in any programming languagethat can be executed on a client system and/or server or server systemsuch as, for example, C, C++, HTML, any other markup language, Java™,JavaScript, ActiveX, any other scripting language, such as VBScript, andmany other programming languages as are well known may be used. (Java™is a trademark of Sun Microsystems, Inc.).

According to one embodiment, each system 416 is configured to providewebpages, forms, applications, data and media content to user (client)systems 412 to support the access by user systems 412 as tenants ofsystem 416. As such, system 416 provides security mechanisms to keepeach tenant's data separate unless the data is shared. If more than oneMTS is used, they may be located in close proximity to one another(e.g., in a server farm located in a single building or campus), or theymay be distributed at locations remote from one another (e.g., one ormore servers located in city A and one or more servers located in cityB). As used herein, each MTS could include one or more logically and/orphysically connected servers distributed locally or across one or moregeographic locations. Additionally, the term “server” is meant toinclude a computer system, including processing hardware and processspace(s), and an associated storage system and database application(e.g., OODBMS or RDBMS) as is well known in the art. It should also beunderstood that “server system” and “server” are often usedinterchangeably herein. Similarly, the database object described hereincan be implemented as single databases, a distributed database, acollection of distributed databases, a database with redundant online oroffline backups or other redundancies, etc., and might include adistributed database or storage network and associated processingintelligence.

FIG. 5 also illustrates environment 410. However, in FIG. 5 elements ofsystem 416 and various interconnections in an embodiment are furtherillustrated. FIG. 5 shows that user system 412 may include processorsystem 412A, memory system 412B, input system 412C, and output system412D. FIG. 5 shows network 414 and system 416. FIG. 5 also shows thatsystem 416 may include tenant data storage 422, tenant data 423, systemdata storage 424, system data 425, User Interface (UI) 530, ApplicationProgram Interface (API) 532, PL/SOQL 534, save routines 536, applicationsetup mechanism 538, applications servers 500 ₁-400 _(N), system processspace 502, tenant process spaces 504, tenant management process space510, tenant storage area 512, user storage 514, and application metadata516. In other embodiments, environment 410 may not have the sameelements as those listed above and/or may have other elements insteadof, or in addition to, those listed above.

User system 412, network 414, system 416, tenant data storage 422, andsystem data storage 424 were discussed above in FIG. 4. Regarding usersystem 412, processor system 412A may be any combination of one or moreprocessors. Memory system 412B may be any combination of one or morememory devices, short term, and/or long term memory. Input system 412Cmay be any combination of input devices, such as one or more keyboards,mice, trackballs, scanners, cameras, and/or interfaces to networks.Output system 412D may be any combination of output devices, such as oneor more monitors, printers, and/or interfaces to networks. As shown byFIG. 5, system 416 may include a network interface 420 (of FIG. 4)implemented as a set of HTTP application servers 500, an applicationplatform 418, tenant data storage 422, and system data storage 424. Alsoshown is system process space 502, including individual tenant processspaces 504 and a tenant management process space 510. Each applicationserver 500 may be configured to tenant data storage 422 and the tenantdata 423 therein, and system data storage 424 and the system data 425therein to serve requests of user systems 412. The tenant data 423 mightbe divided into individual tenant storage areas 512, which can be eithera physical arrangement and/or a logical arrangement of data. Within eachtenant storage area 512, user storage 514 and application metadata 516might be similarly allocated for each user. For example, a copy of auser's most recently used (MRU) items might be stored to user storage514. Similarly, a copy of MRU items for an entire organization that is atenant might be stored to tenant storage area 512. A UI 530 provides auser interface and an API 532 provides an application programmerinterface to system 416 resident processes to users and/or developers atuser systems 412. The tenant data and the system data may be stored invarious databases, such as one or more Oracle™ databases.

Application platform 418 includes an application setup mechanism 538that supports application developers' creation and management ofapplications, which may be saved as metadata into tenant data storage422 by save routines 536 for execution by subscribers as one or moretenant process spaces 504 managed by tenant management process 510 forexample. Invocations to such applications may be coded using PL/SOQL 534that provides a programming language style interface extension to API532. A detailed description of some PL/SOQL language embodiments isdiscussed in commonly owned U.S. Pat. No. 7,730,478 entitled, “Methodand System for Allowing Access to Developed Applicants via aMulti-Tenant Database On-Demand Database Service”, issued Jun. 1, 2010to Craig Weissman, which is incorporated in its entirety herein for allpurposes. Invocations to applications may be detected by one or moresystem processes, which manage retrieving application metadata 516 forthe subscriber making the invocation and executing the metadata as anapplication in a virtual machine.

Each application server 500 may be communicably coupled to databasesystems, e.g., having access to system data 425 and tenant data 423, viaa different network connection. For example, one application server 500₁ might be coupled via the network 414 (e.g., the Internet), anotherapplication server 500 _(N-1) might be coupled via a direct networklink, and another application server 500 _(N) might be coupled by yet adifferent network connection. Transfer Control Protocol and InternetProtocol (TCP/IP) are typical protocols for communicating betweenapplication servers 500 and the database system. However, it will beapparent to one skilled in the art that other transport protocols may beused to optimize the system depending on the network interconnect used.

In certain embodiments, each application server 500 is configured tohandle requests for any user associated with any organization that is atenant. Because it is desirable to be able to add and remove applicationservers from the server pool at any time for any reason, there ispreferably no server affinity for a user and/or organization to aspecific application server 500. In one embodiment, therefore, aninterface system implementing a load balancing function (e.g., an F5Big-IP load balancer) is communicably coupled between the applicationservers 500 and the user systems 412 to distribute requests to theapplication servers 500. In one embodiment, the load balancer uses aleast connections algorithm to route user requests to the applicationservers 500. Other examples of load balancing algorithms, such as roundrobin and observed response time, also can be used. For example, incertain embodiments, three consecutive requests from the same user couldhit three different application servers 500, and three requests fromdifferent users could hit the same application server 500. In thismanner, system 416 is multi-tenant, wherein system 416 handles storageof, and access to, different objects, data and applications acrossdisparate users and organizations.

As an example of storage, one tenant might be a company that employs asales force where each salesperson uses system 416 to manage their salesprocess. Thus, a user might maintain contact data, leads data, customerfollow-up data, performance data, goals and progress data, etc., allapplicable to that user's personal sales process (e.g., in tenant datastorage 422). In an example of a MTS arrangement, since all of the dataand the applications to access, view, modify, report, transmit,calculate, etc., can be maintained and accessed by a user system havingnothing more than network access, the user can manage his or her salesefforts and cycles from any of many different user systems. For example,if a salesperson is visiting a customer and the customer has Internetaccess in their lobby, the salesperson can obtain critical updates as tothat customer while waiting for the customer to arrive in the lobby.

While each user's data might be separate from other users' dataregardless of the employers of each user, some data might beorganization-wide data shared or accessible by a plurality of users orall of the users for a given organization that is a tenant. Thus, theremight be some data structures managed by system 416 that are allocatedat the tenant level while other data structures might be managed at theuser level. Because an MTS might support multiple tenants includingpossible competitors, the MTS should have security protocols that keepdata, applications, and application use separate. Also, because manytenants may opt for access to an MTS rather than maintain their ownsystem, redundancy, up-time, and backup are additional functions thatmay be implemented in the MTS. In addition to user-specific data andtenant specific data, system 416 might also maintain system level datausable by multiple tenants or other data. Such system level data mightinclude industry reports, news, postings, and the like that are sharableamong tenants.

In certain embodiments, user systems 412 (which may be client systems)communicate with application servers 500 to request and updatesystem-level and tenant-level data from system 416 that may requiresending one or more queries to tenant data storage 422 and/or systemdata storage 424. System 416 (e.g., an application server 500 in system416) automatically generates one or more SQL statements (e.g., one ormore SQL queries) that are designed to access the desired information.System data storage 424 may generate query plans to access the requesteddata from the database.

Each database can generally be viewed as a collection of objects, suchas a set of logical tables, containing data fitted into predefinedcategories. A “table” is one representation of a data object, and may beused herein to simplify the conceptual description of objects and customobjects. It should be understood that “table” and “object” may be usedinterchangeably herein. Each table generally contains one or more datacategories logically arranged as columns or fields in a viewable schema.Each row or record of a table contains an instance of data for eachcategory defined by the fields. For example, a CRM database may includea table that describes a customer with fields for basic contactinformation such as name, address, phone number, fax number, etc.Another table might describe a purchase order, including fields forinformation such as customer, product, sale price, date, etc. In somemulti-tenant database systems, standard entity tables might be providedfor use by all tenants. For CRM database applications, such standardentities might include tables for Account, Contact, Lead, andOpportunity data, each containing pre-defined fields. It should beunderstood that the word “entity” may also be used interchangeablyherein with “object” and “table”.

In some multi-tenant database systems, tenants may be allowed to createand store custom objects, or they may be allowed to customize standardentities or objects, for example by creating custom fields for standardobjects, including custom index fields. U.S. patent application Ser. No.10/817,161, filed Apr. 2, 2004, entitled “Custom Entities and Fields ina Multi-Tenant Database System”, and which is hereby incorporated hereinby reference, teaches systems and methods for creating custom objects aswell as customizing standard objects in a multi-tenant database system.In certain embodiments, for example, all custom entity data rows arestored in a single multi-tenant physical table, which may containmultiple logical tables per organization. It is transparent to customersthat their multiple “tables” are in fact stored in one large table orthat their data may be stored in the same table as the data of othercustomers.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the invention. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment.

While the invention has been described in terms of several embodiments,those skilled in the art will recognize that the invention is notlimited to the embodiments described, but can be practiced withmodification and alteration within the spirit and scope of the appendedclaims. The description is thus to be regarded as illustrative insteadof limiting.

What is claimed is:
 1. A method of managing data in a relationaldatabase environment and a non-relational database environment, themethod comprising: identifying data in the relational databaseenvironment that is static and to be maintained beyond a preselectedthreshold length of time with one or more server computing devices;copying the data from the relational database; storing the data thenon-relational database; and providing access to the data from thenon-relational database via a user interface that accesses both therelational database and the non-relational database.
 2. The method ofclaim 1 wherein the data comprises field history data.
 3. The method ofclaim 1 wherein the relational database environment and thenon-relational database environment are part of a multitenantenvironment.
 4. The method of claim 1 wherein the copying the data fromthe relational database to the non-relational database comprises:receiving a key prefix to identify the data to be copied; splitting thedata to be copied into at least a first batch and a second batch;copying a first batch of data from the relational database to thenon-relational database; automatically, in response to copying the firstbatch, enqueuing a second batch of data to be copied from the relationaldatabase to the non-relational database.
 5. The method of claim 1wherein identifying data in the relational database environment that isstatic and to be maintained beyond a preselected threshold length oftime is performed in response to detection of one or more triggerconditions within the relational database environment.
 6. A system formanaging data in a relational database environment and a non-relationaldatabase environment, the system comprising: at least one servercomputing device to provide services to one or more remote clientcomputing devices; a database management environment to support servicesprovided by the server computing device, the database managementenvironment to identify data in the relational database environment thatis static and to be maintained beyond a preselected threshold length oftime with one or more server computing devices, to copy the data fromthe relational database, to store the data the non-relational database,and to provide access to the data from the non-relational database via auser interface that accesses both the relational database and thenon-relational database.
 7. The system of claim 6 wherein the datacomprises field history data.
 8. The system of claim 6 wherein therelational database environment and the non-relational databaseenvironment are part of a multitenant environment.
 9. The system ofclaim 6 wherein the copying the data from the relational database to thenon-relational database comprises: receiving a key prefix to identifythe data to be copied; splitting the data to be copied into at least afirst batch and a second batch; copying a first batch of data from therelational database to the non-relational database; automatically, inresponse to copying the first batch, enqueuing a second batch of data tobe copied from the relational database to the non-relational database.10. The system of claim 6 wherein identifying data in the relationaldatabase environment that is static and to be maintained beyond apreselected threshold length of time is performed in response todetection of one or more trigger conditions within the relationaldatabase environment.
 11. A non-transitory computer-readable mediumhaving instructions stored thereon to manage data in a relationaldatabase environment and a non-relational database environment, theinstructions, when executed by one or more processors, to: identify datain the relational database environment that is static and to bemaintained beyond a preselected threshold length of time with one ormore server computing devices; copy the data from the relationaldatabase; store the data the non-relational database; provide access tothe data from the non-relational database via a user interface thataccesses both the relational database and the non-relational database.12. The non-transitory computer-readable medium of claim 11 wherein thedata comprises field history data.
 13. The non-transitorycomputer-readable medium of claim 11 wherein the relational databaseenvironment and the non-relational database environment are part of amultitenant environment.
 14. The non-transitory computer-readable mediumof claim 11 wherein the copying the data from the relational database tothe non-relational database comprises: receiving a key prefix toidentify the data to be copied; splitting the data to be copied into atleast a first batch and a second batch; copying a first batch of datafrom the relational database to the non-relational database;automatically, in response to copying the first batch, enqueuing asecond batch of data to be copied from the relational database to thenon-relational database.
 15. The non-transitory computer-readable mediumof claim 11 wherein identifying data in the relational databaseenvironment that is static and to be maintained beyond a preselectedthreshold length of time is performed in response to detection of one ormore trigger conditions within the relational database environment. 16.An apparatus for managing data in a relational database environment anda non-relational database environment, the method comprising: means foridentifying data in the relational database environment that is staticand to be maintained beyond a preselected threshold length of time withone or more server computing devices; means for copying the data fromthe relational database; means for storing the data the non-relationaldatabase; means for providing access to the data from the non-relationaldatabase via a user interface that accesses both the relational databaseand the non-relational database.
 17. The apparatus of claim 16 whereinthe data comprises field history data.
 18. The apparatus of claim 16wherein the relational database environment and the non-relationaldatabase environment are part of a multitenant environment.
 19. Theapparatus of claim 16 wherein the means for copying the data from therelational database to the non-relational database comprises: means forreceiving a key prefix to identify the data to be copied; means forsplitting the data to be copied into at least a first batch and a secondbatch; means for copying a first batch of data from the relationaldatabase to the non-relational database; means for automatically, inresponse to copying the first batch, enqueuing a second batch of data tobe copied from the relational database to the non-relational database.20. The apparatus of claim 16 wherein identifying data in the relationaldatabase environment that is static and to be maintained beyond apreselected threshold length of time is performed in response todetection of one or more trigger conditions within the relationaldatabase environment.